Keith Packard: calypso
Calypso CalDAV/CardDAV/WebDAV for Android and Evolution
Ever since I bought my first Palm Pilot in 1997, I ve relied upon a
pocket-able device to carry a copy of my calendar and contacts, and for
that same database to be present on my laptop. I went through a long
list of Palm-compatible devices, including both the Palm Treo and Palm
Centro telephones. I even wrote a number of my own Palm
applications. Years ago, it was pretty obvious that I d have
to find a new phone, but I was stuck looking for something that could
provide the same hot-sync functionality.
SyncML on the Series 40
I bought a Series 40 Nokia phone in Shanghai that promised SyncML
support. Given that I had seen numerous SyncML implementations for
Linux, it seemed like I should be able to get something to work.
SyncML on the Series 40 is a disaster the phone couldn t actually
store all of my contacts, and it couldn t hold half of the data
fields I used. So, that phone landed in the box and back to the Centro
I went.
SyncML on the N900
Nokia kindly sent me an N900 at some point, so I gave SyncML another
try. Given that the N900 runs evolution-data-server, and that I ve had
evolution-data-server running on my laptop, it seemed like I should be
in business. Well, almost. It took several days of hacking to fix bugs
in the evolution SyncML back end, then another several days fussing
with opensync.
I ended up abandoning direct synchronization as unworkable opensync
would sit in an infinite loop, or worse, trash my database
completely. I finally found syncml-ds-tool , which is a debugging
tool that comes with opensync. This tool simply synchronizes a set of
disk files, one per contact or calendar entry, with the phone. That
worked for well over a year. And then, a few months ago, Bluetooth on
the laptop stopped connecting with the phone s SyncML server. I d get
ECONNREFUSED every time I tried to use it. So much for the N900. DUN
still worked, mostly, although it too would get ECONNREFUSED at times,
but retrying seemed to make it work.
However, While the N900 SyncML solution worked, I discovered another
thing I wanted contacts and calendar entries stored in individual
files and revision controlled with git. This makes it reasonable to
delete stale calendar entries and know that they re never really gone,
just left behind in an older version of the calendar. And, if you mess
up, you can recover by poking at the database with git directly.
I switched back to the venerable Palm Centro; it turns out that
calendar and contacts are more important to me than being able to surf
the web on my phone. Alas, my Centro went swimming in August and has
passed on to the great electronics recycling house in the sky. I pulled
the SIM out and switched back to the N900.
I got my contacts imported on the N900 by copying files over the net
work; not a long-term strategy, but at least I had phone numbers
again. There was no hope for my calendar. I started looking for a
solution in earnest.
How about Android?
At this point in my story, I m sure you re asking why I didn t just
use one of the numerous Android phones that came through my hands. The
answer is simple my calendar and contacts are probably some of my
most personal data, and I m not willing to store them outside of my
direct control, for reasons similar to those which are driving the
development of the FreedomBox.
When Android first came out, it could only talk to Google
services, which didn t meet my hard requirement for personal data
storage.
One of my co-workers had his Google account suspended for violating
the terms-of-service; he asked what he had done, but they wouldn t
say. He asked if he could get his data back, and they said no . They
invited him to create a new account, but it would not ever get any of
the old data. A few days later, he got a nice apologetic email letting
him know that they d made a mistake and that he hadn t, in fact,
violated any of the terms-of-service, and that his old account was
restored with all of his data intact. Wasn t that nice of Google?
WebOS?
I got a WebOS phone over the summer and discovered that while it had
multiple contact/calendar back ends, none of them used standard
protocols and so you only had the choice between multiple corporate
data centers, which isn t actually a choice at all.
Furthermore, the WebOS phone refused route PAN packets over the phone
network, even though I have a data plan which allows this. It s not
that it couldn t support it, it s that it refused.
A couple of weeks after the WebOS phone arrived, HP canceled all of
their WebOS hardware products, which made me less interested in trying
to solve this problem.
Android recovers
About the same time the WebOS phone arrived, I discovered that Google
had published enough information about the calendar/contacts internals
for Marten Gajda to write CalDAV-Sync and
CardDAV-Sync. And then, Andrew McMillan
wrote aCal, which is a
complete replacement for the built-in calendar and contacts
applications and supports CalDAV and CardDAV.
With two different standards-compliant solutions available, it seemed
like it might be time to try Android again.
I d love for CardDAV-Sync and CalDAV-Sync to become free software like
aCal is. Andrew makes money from aCal by offering it for sale via the
Android Market, while still publishing the sources for those who want
to build their own copy.
CalDAV/CardDAV on Linux
I think the most widely known CalDAV server for Linux is probably
DAViCal, a huge pile of PHP and SQL sitting
on top of Apache. I m sure it s suitable for running on a server
and being accessed over the internet, but I m not interested in that,
nor am I interested in having my laptop run Apache and PostgreSQL.
I found a tiny little CalDAV server, Radicale,
which seemed like a lot better fit. It s written in Python and uses
the usual Python HTTP server infrastructure, which provides SSL and
authentication support along with some fairly convenient APIs for
parsing and generating HTML.
Before long, I discovered that Radicale was actually too simple for
my needs. It stores the whole calendar in a single file, re-parsing it
whenever a request is made, so a calendar with just hundreds of
entries caused the server to slow down enough that evolution would
time-out when talking to it.
Also, Radicale doesn t actually parse the calendar entries completely,
it has some ad-hoc code that finds various pieces of data, but without
dealing with the whole syntax.
I started hacking at Radicale to see how far I could get. I changed
the storage code to store one event per file, then added hooks to use
git for change management. Then, I found a full vcalendar/vcard
parsing library in python,
vobject, which I used to
replace the ad-hoc parsing code. Finally, I added support for VCARD
entries as well, allowing the system to store both calendar and
contact information.
Introducing Calypso, a CardDAV/CalDAV server
With this much divergence from the original project, I ve figured I d
best rename things to avoid confusion, so I decided to call it
calypso , after a brief trip through the dictionary looking for names
starting with ca .
Calypso works with evolution, iceowl and the Android CalDAV/CardDAV
plugins. It does not yet work with aCal; for some reason aCal cannot
find any calendars on the server.
Calypso also supports importing calendar changes from the command
line, allowing you to integrate support into a text-based email
application like notmuch or
mutt.
Calypso is available via git from git://keithp.com/git/calypso and is
distributed under the GPL (v3 or later). I still consider it a work
derived from Radicale, and so the code retains all of the Radicale
copyrights along with my own.
Using Calypso
Initial setup
Calypso runs as a regular user, all data are stored in
~/.config/calypso. To initialize calypso:
$ mkdir ~/.config/calypso ~/.config/calypso/calendars
$ cat > ~/.config/calypso/config << EOF
[server]
ssl=true
certificate=/home/keithp/.config/calypso/ssl/server.crt
key=/home/keithp/.config/calypso/ssl/server.key
[acl]
;type=htpasswd
type=fake
filename=/home/keithp/.config/calypso/passwd
Running calypso
Then run calypso:
$ python ./calypso.py
No, I haven t figured out how to install it
Creating new calendars
To add a new database:
$ mkdir -p ~/.config/calypso/calendars/private/my_calendar
$ cd ~/.config/calypso/calendars/private/my_calendar
$ git init
$ git commit --allow-empty -m'initialize new calendar'
The new calendar should now be visible as
https://localhost:5233/private/my_calendar
You can add files to the directory at any time; calypso will check the
directory mtime at each operation and update its internal state from
that on disk automatically when the directory changes.
Importing files
Given a set of files with VCALENDAR or VCARD entries, you can import
them with:
$ calypso --import private/my_calendar <filenames...>
This will update any changed entries and add any new ones.
ToDo list for calypso
- Document the config file contents.
- Make it installable
- Figure out what aCal wants
- Support calendar creation
- Synker. This provides a desktop widget which starts the sync process running on a list of accounts. This makes it easy to manually synchronize with the laptop when you are connected
- Contact Editor. This lets you fully edit contacts synchronized over CardDAV. The built-in contact editor doesn t let you change anything other than the name for some reason.